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ABSTRACT 

Courseware designers aim to innovate information communications technology (ICT) tools to increase learning 
experiences, spending many hours developing eLearning programmes. This effort gives rise to a dynamic technological 
pedagogical environment. However, it is difficult to recognise whether these online programmes reflect an instructional 
design (ID)-model or whether they can be substantiated through sound ID principles. This study presents a systematic 
courseware design-validation procedure; giving preliminarily empirical results from learners’ cognitive performance 
outcomes. A series of 2x3 factorial quasi -experiments were conducted to validate the performance instrumentation and to 
substantiate the effectiveness of the proposed courseware -design model. A total of 167 -participants, from four higher 
education institutions took part in this research project. Participants’ cognitive preferences were identified using the 
cognitive style analysis (CSA) test. Initial observations suggest that testing instruments were able to make reliable 
probabilistic inferences of the cognitive performance outcomes. 
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1. INTRODUCTION 

Since the inception of the instructional design (ID) discipline by Robert Gagne (1985), many research studies 
since then have investigated how to improve the instructional environments and learning experiences that 
promote the acquisition of specific knowledge and skills (Merrill et al. 1996). The literature reveals the 
proliferation of such ID-models among various schools of thought. The most widely applied models include: 
the generic ADDIE (1975); the Hoffman and Ritchie’s ICARE (1998); the Dick and Carey (1978); and 
Heinich et al’s ASSURE (1996). However, further reviews pinpoint the limitations of existing ID-models as 
being ineffective and mainly developed to guide the practice of specific tasks (Young 2008). It seems 
possible that these conclusions can be contributed to the lack of empirical evidence and rigorous ratification 
processes to measure the effectiveness of these ID-models under different instructional environments as 
claimed by Branch and Kopch (2014). Thus, the main objective of this paper is to describe the ID -design 
process used to develop an eTutorial courseware module and the calibration of the testing instruments used to 
examine the expected changes in the knowledge acquisition following the learners’ courseware participation. 
This is an initial paper in a series of papers which are planned to describe a doctoral research study. The 
structure of the paper commences with an overview of ID-models, then it presents the prescriptive ID-model 
adopted for this work. Next it briefly describes the design and development of the eTutorial courseware 
module. The doctoral methodology sections include: the experimental procedures carried out, and the 
development of the testing instrumentation. The preliminary findings are then presented relating to the 
validation of the testing instrumentation; the paper closes with a conclusion/discussion. 
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2. ID-MODELS 

Currently, there are scant contributions in the literature from the instructional systems design (ISD) 
community that is applicable for Web 2.0 instructional-media. For instance; there is: the Dick & Carey 
1970's Model (revised by them in 2004); the 4C-ID model (van Merrienboer, Jelsma & Paas (1992), then 
revised by van Merrienboer & Jochems (2004); and the ASSURE model devised by Heinich et al, (1996). 
Instead the researchers use familiar ID-models that were designed for use before the advances that 
multimedia now offers courseware design. We believe therefore, that this paper is focused on the collision 
point concerning the use of long established ID-models to represent the pedagogical development of the ISD 
for online courseware with the novel approach that such technological advances deserve. Culatta (2013) has 
published a list of commonly accepted prescriptive ID-models that use commonly known ID-specifications 
or theoretical frameworks to advance the creation of instructional programmes. However, through closer 
scrutiny of this list of 25 such pedagogical models, there are only 14 that focus on ISD that have been 
updated for our modern computing environment. And so, one of the purposes of this study was to bring 
forward the evidence for updating the ISD community with our prescriptive ID -model, through our 
courseware design-evaluation techniques. 

2.1 Substantiation of the Prescriptive ID-model 

The prescriptive ID-model used in this study involved key ID elements, including: analysis (of the 
instructional content); design (of the ePedagogical content and assessment strategies); development (of the 
online IS -artefacts); implementation of the instructional programme and evaluation of the effectiveness of the 
instructional outcomes) (figure 1). This ID-model extends the Branson et al. (1975) ADDIE model through 
the systematic examination of the participants’ performance outcomes. Our ID-model is based on key 
activities involving: planning out the required change in the instructional environment; executing the 
methodology; observing the results; analysing the subsequent data; refining the test-items; recording the 
results; and reflecting on the subsequent outcomes. We propose that the systematic running of the successive 
studies together with the continual data analysis of the performance outcomes provides the empirical 
evidence to substantiate the effectiveness of this ID-model. 



Figure 1. Prescriptive ID- model adapted from ADDIE, Branson et al. (1975) 


3. THE DESIGN OF THE E-TUTORIAL MODULE 

The learning content and lesson-plan used to develop the ePedagogical materials were exactly the same in the 
three educational environments (T1 face-to-face (traditional classroom facilitation), T2 blended (combination 
of traditional classroom facilitator-led and computerised interaction), and T3 wholly computerised (online) 
interaction). The instructional objectives of the eTutorial courseware module were designed to enable the 
students (with the help of information communications technology (ICT) tools) to identify data-flow 
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diagrams (DFD) levels, thereby constituting a complete DFD set. Several key information systems 
(IS)-design activities were undertaken, such as: storyboarding and system testing, were conducted during the 
design and development of the eTutorial courseware module. Key ID principles were adopted to promote 
effective learning and to optimise knowledge acquisition. Some influential ID elements were used, such as: 

• Home page: The eTutorial started with a 'welcoming page' aimed to introduce learners to the topic and 
how the structure of the overall module (Clark & Mayer, 2008). To conceptualize the abstract notion 
(Merrill & Tennyson, 1977) of DFD levels, the module was designed depicting a multi-levelled business 
building with the context diagram as the Ground Level leading to other levels (see Figure 2). 

• Instructions page: The instructions page (Figure 3) was designed to inform learners of how the 
eTutorial module worked and the interactivity features they could use when starting the tutorial 
(Knowlton & Simms, 2010). And so, the eTutorial was divided into four main parts: context diagram; 
level-0 diagram; level- 1 diagram; concluding with quiz activities (providing immediate feedback). 

• Self-paced orientation: Prompting self-paced eTutorial browsing enabling self-timed instruction. 

• Interactivity features: The eModule offered a range of interactive elements which were specifically 
designed to align with different cognitive styles/instructional mode preferences (for verbal/pictorial 
instructional media) and to promote enhanced learning opportunities. For example, navigation bars were 
available in two locations on the computer-screen. The one at the bottom of the screen was designed for 
Analytic users to allow smooth movement among the module parts since they tend to view content as a 
connected parts focusing on one part or two at a time. Whereas the knowledge-navigator bar on the left- 
side of the screen may attract the Wholists, who cognitively process their information in an overall 
manner; thereby providing users with the option to skip, repeat, or choose certain parts of the lesson 
(figure4). 

• Presentation of materials: The instructional content was presented in textual blocks, associated 
pictures, and diagram-mode in order to facilitate acquisition of knowledge for the Imagers who prefer 
pictorial representation; while the Verbalisers learn faster from text-based materials (Riding & Rayner, 
1998 McKay, 2002). Colors were used to highlight critical parts of the lesson. For instance, Wholist 
participants are more likely unable to make clear distinctions among different ePedagocical-parts due to 
their (inherent) holistic processing preference (Riding & Cheema, 1991). Similarly, Analytic preferenced 
learners may overlook integral parts as they focus on one or two ePedagogical-parts as they ‘think’ about 
the information they are receiving (Riding, 2001; McKay, 2000). Thus, colors were employed to draw in 
both cognitive-preferenced groups’ attention. In addition, a click button sign was located next to 
instructional content to provide extra knowledge for the learners who would like to gain more information 
on a topic or a concept. 



Figure 2. The homepage of the e-tutorial module 


Figure 3. Instructions page of the e-module 



Figure 4. Interactivity features 
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4. METHODOLOGY 
4.1 Participants 


A total of 167 undergraduates, who were officially enrolled in the 'Information System Analysis and Design' 
(ISA&D) course at four public HE institutions, volunteered for this study and took part in different stages of 
the experiments. Two months prior to the main experiment, participants underwent the cognitive styles 
analysis (CSA) (Riding & Cheema, 1991), a computerised assessment test used to identify their cognitive 
preferences. The CSA results (figure 5) were recorded in an Excel spreadsheet, and used to randomly assign 
participants into one of the three instructional delivery mode groups. Figure 6 is an illustration of 
participants’ allocation process based on their CSA results. The blue triangle represented participants in the 
conventional instructor led/ F-2-F group, the green diamonds depicted the computerised group, and the red 
square represented the blended group (classroom instructor and parts of the computerised module). 
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Figure 5. CSA results 



Figure 6. An illustration of participants’ allocation 


4.2 Experimental Research Procedure 

The experimental approach, adopted a series of 2x3 factorial quasi experiments that were conducted for 
separate sequential studies that were planned for this research project. The staged quasi-experiment (figure 7) 
was carefully designed to start with a formal registration process during the first step, followed by a short 
verbal explanation of the research schedule, after which participants underwent a pre-test (the first 
instrumentation assessment designed to assess their ‘entry’ knowledge prior to the instructional intervention). 
Then participants were randomly allocated, by the researcher, into one of the three instructional delivery 
modes (Tl, T2 or T3 mentioned earlier) using their CSA results. Next was the instructional intervention in 
which each group took the same instructional content in different modes, as Tl, T2 or T3. The final step was 
the post-test (the second instrumentation assessment aimed to measure the participants’ change of knowledge 
after the intervention). All participants underwent the experiments under comparable conditions. For 
instance, they received the same lecture instructional content without prior knowledge of the topic under 
different delivery modes Tl, T2 or T3) within the same timeframe, and were assessed with the same 
assessment tools (pre- and post-tests). 
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Figure 7. Experimental procedure 


4.3 Research Instruments 

4.3.1 Instructional Materials 

A hierarchal task analysis based on the Gagne (1985) hierarchal task analysis approach, was conducted to 
narrow down the pedagogical focus. Process Modeling with DFD techniques was the chosen 
instructional-lesson in an ISA&D undergraduate course. The output from this activity was the identification 
of entry-level skills and required skills to successfully achieve the instructional objectives. The experimental 
lesson plan was then prepared based on the Gagne conditions of learning theory (1985) and the Reigeluth 
(1983) elaboration theory. Next was the design of a 'skills development matrix' which guided the data 
collection and analysis process, adapted from Mat-Jizat (2012); McKay (2000). The horizontal axis of this 
matrix was designed based on the type of analysis knowledge (commencing with declarative then procedural 
attributes) and the six intellectual learning categories identified by Gagne (1985). The vertical axis 
represented the required ISA&D skills identified necessary for a participant to achieve the instructional 
objectives that were plotted according to their difficulty level starting at the point of origin on the matrix, 
with the easier skills to the more difficult ones. 

4.3.2 The Pre-and-Post-Tests 

The pre-and-post-tests formed the main assessment instrumentation. Thus, it was necessary to follow the 
Izard’s (2005) systematic approach in constructing test-items. This step was critical in order to ensure that the 
test-items would provide meaningful evidence by which to make reliable inferences. As for the scoring 
technique, participants’ raw scores for each test-item were converted into numeric values to align with the 
QUEST analysis software. Dichotomous and partial credits were the main scoring categories. Answers from 
dichotomous test-items were recorded as a 'O' or a T,' whereas partial credit items were recorded with a '0,' T,' 
or a '2.' Participants’ scores were documented in a data table and saved as a data file. 


5. PRELIMINARILY FINDINGS 

Results for this paper include the preliminarily analysis of a series of a particular data set, as declaring the 
full data for this research project is beyond the scope of this paper. Therefore, these data must be interpreted 
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with caution. The QUEST interactive test analysis system (Adams & Khoo, 1996), was built based on a 
Rasch model (Rasch,1993) and the Item Response Theory (IRT), was used to analyse the data. It generated 
results such as: fit statistics; and test-item and case (participant) estimates, in the form of maps and 
statistical-tables. Some of the empirical data below is at its first iteration, which with subsequent analysis 
runs will provide a deeper analysis for this study’s variables. 


5.1 Validity of Assessment Instrumentations 


It was necessary to first calibrate testing instrumentation as implications/conclusions depend on the 
assessment instruments’ assured validity and reliability. During our prescriptive ID-model’ s design phase, 
there were four separate studies that were conducted to align with its aim as the formal calibration of the 
testing instrumentation. Results from these first two studies were used to monitor the behaviour of test-items, 
and to delete or substitute unsuitable test-items for subsequent studies. Thus, many adjustments were made 
prior to data collection for the main study. The QUEST item fit map (table 1) provides a visual representation 
of the magnitude of the fit statistic of test-items that were conforming to the Rasch requirements. Table 2 
represents the item fit maps of pre-and-post-tests from one study conducted for this research project. Each 
test-item is represented by a star and should lie within the dotted lines (thresholds), which define the 
acceptable range of good behaving test-items. Unreliable test items lie outside the dotted lines. For example, 
test-items 12 and 22 on table l.a, cannot be considered conforming to the Rasch model because it is not 
behaving in a consistent manner to other test-items. Table l.b shows all test-items lie between the dotted line, 
fitting the Rasch model, and therefore deemed reliable. 


Table l.a. Item Fit map (Pre-Test) Table l.b. Item Fit map (Post-Test) 
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5.2 Participants and Test-items’ Performance Indices 

The QUEST variable map provides measurement indices of participants’ performance relative to other 
participants, and relative to test-items. It enables the performance evaluation of participants and test-items 
simultaneously on a uni-dimensional logit scale. The variable maps of post-tests from the four studies 
conducted for this research project are shown in figure 8. The left-side of each map shows the distribution of 
participants’ abilities based on their performance (each X represents a participant). And so, the performance 
of participants on the upper left of the maps is better than participants at the lower left of the map. Test-items 
are plotted on the right-side of the map, based on their difficulty level in an easiest-bottom to hardest-top 
sequence. Thus, test-items positioned on the lower right deemed too easy to challenge participants’ abilities 
whereas items on the upper right were beyond participants’ abilities -level. 

Looking at these maps, and as mentioned previously that Study 1 and 2 aimed to test the reliability of 
test-items on a small population (15, and 52 participants respectively) prior to conducting further studies. 
These initial observations required improvements which included the addition of test-items targeting the 
measurement of higher skills on the skills matrix (table #). We tested to the reliability of the ‘new/added’ 
test-items by give the test for a larger population as to improve the precision of test-item measures. And so, 
the subsequent studies had larger well-targeted population (91 participants), which provided better 
information. Variable maps for Study 3 and 4, shows better test-items/people’s performance distribution. 
Calibration carried out on the sample for previous studies, validated the construct scaling as to facilitate the 
acquisition of skills of ‘Data Modeling with DFDs,’ from the easiest into the hardest sequence. Data for study 
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4 was a combination of dichotomous and partial credit scored test-items, and participants were mostly 
clustered between 0 and 3 logits (very few were located outside that). Any participant with ability below 0 
logit and beyond 3 logit was excluded from analysis, because there were not enough test -items to match their 
estimated achievement level. Test-items that did not test any participant’s ability were also discarded, as 
there were not enough participants to measure its reliability. Although, data from these sequential studies, at 
its first iteration, test instruments provided sufficient adherence to the Rasch measurement requirements. 
Therefore, it can be considered reliable to conduct a deeper analysis to measure the cognitive performance. 
Further analysis for this research project is planned to model data from other QUEST estimate outputs in 
order to investigate and compare the performance of different groups involved based on their cognitive 
preference (Wholist- Analytic and Verbaliser-Imagery), or delivery mode (F-2-F, blended, and computerised). 



6. CONCLUSION 

Instructional systems (courseware) designers are using traditional ID -models for their development of the 
online instructional programmes. Consequently with the advent of the more powerful Web 2.0 multimedia 
tools, knowing which the best ID-framework to use can be a vexing issue, especially when considering 
instructional systems design. There are ID-models that people use, however it has been suggested in this 
paper that few of these are suited for adoption in 2016 and onwards. Consequently, the need for development 
of new prescriptive ISD-models is apparent. To this end the prescriptive ID-model used in the research 
described in this paper reflects the call from the literature for such IS -design innovation. 

We set out to explore the effectiveness of adopting the ADDIE model, thereby bringing it forward as an 
ID-model exemplar for ISD practices, in the first instance and as necessary evidence supported by a 
comprehensive data analysis of the participants' performance outcomes. To this end we operationalised the 
research variables (instructional treatments represented by face-to-face as the traditional classroom 
facilitation, a blend of the traditional facilitator-led and computerised instructional activities; and the wholly 
computerized online approach, in an empirically design set of experiments. The primary aim of our research 
was to investigate the interactivity of the instructional treatments and the participants' cognitive information 
processing characteristics, using the cognitive style analysis (CSA) test that we used to allocate participants 
randomly to their instructional treatment groups. We have reported on the initial data analysis in this paper as 
a forerunner to the outcomes from the main experiment. Already we are able to show there were changes in 
knowledge outcomes on data flow diagramming techniques. 
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